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DESCRIPTION 
Picture Generating Apparatus and Picture Generating Method 
Technical Field 

This inventionrelates to apicture generating apparatus and apicture generating 
method for generating distance picture and variable-density (gradation) picture by 
employing the stereo method, and relates to a picture generating apparatus and a 
picturegeneratingmethodforgeneratingdistancepicture indicating distancebetween 
virtualpositionandobjecttobehnagedandvariable-density(gradation)picturewhen 
object to be imaged is imaged from virtual position by using two cameras or more. 
Background Art 

As an apparatus using the technique generally called stereo method, there is 
known, e.g., distance measurement unit (apparatus) for generating distance 
information from the camera unit up to object to be imaged. In this distance 
measurement unit, in order to measure three-dimensional coordinate position of the 
surface of object to be imaged, i.e., three dimensional shape thereof, pictures obtained 
by imaging, at the same time, the same object to be imaged by plural camera units 
having different view points are used to determine correspondence points 
corresponding to each other every respective pixels between those pictures to obtain 
distance information from the distance measurement unit up to the object to be 
imaged by that parallax. 

In the prior art, there will be explained the representative first technique for 
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generating distance picture consisting of distance infonnation and v^iabie-density 
picture consisting of lun^inance information, etc. by camera unit disposed at vtrtuai 
position and in virtual direction. At the time of generating distance picture and 
variable-density picture, three-dinrensional measurement is first carried out by 
imaging object to be imaged by means of plural camera units. Tben, spattal 
coordinates (shape) of object to be imaged are determined. Then, an approach ts 

spatia.coordinates(shape)oftheobjecttobeimaged,distancepic«retobeobserved 
bythe camera unit virtually disposed, and to generate variable-densitypicture. 

Explanation will nov. be given in comrection with the second technique m 
which the stereo method is employed to generally determine distance picture w.th 
.espect to object to be imaged, intiris case, explanation wiUbe given in comtectron 

with the example where, for the brevity of explanation, reference camera and 
detection camera for detecting distance between the reference camera and object to 

be imaged arensedtodeterminecorrespondencepointstogeneratedistanceptcture. 

First, such an approach is employed to image (pick up image of) object to be 

taagedbyreferencecameraanddetectioncameradisposedatpositionsdifferentfrom 

eachothertoextractapredeterminedareaftompicmredataobtainedbythereference 

camera to successively shift picmre data of the extracted predetermined area on 
picture of the detection camera. At this time, the picture of the predetermined area 
, shifted on line called epipolar hne determined by internal parameters and/or 
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posiW.e,« ofthedeteaion camera. By s~^^ 
ofp.edete..inedareao„pictureof.heaeteoUo„ camera in this way,pictu^^ 

detection catnera are compared with each other to determine degree of 
correspondence. Ft^er, displacement quantity at displacement position where 
degreeofcorrespondenceisthehighestissetaspa^laxatcentralpixelofthepicture 

oftheextractedpredeterminedarea,lnaddition,byrepeatingtheseprocessingwt«t 
aspect to respectivepixelsofpictureofthereference camera, distancepictures are 

generated. 

Nat^ely, as shown in FIG. 1 , object to be imaged is observed by reference 
camera and detection camera. When it is recognized that point P within the 

possible to determinethree-dimensionalpositionofthepointP.In this case, it.not 
easytoj«dge(discriminate)thatthecorrespondingpointisimagepick-uppomtn,of 

detection camera corresponding to image pick-up point n. of the reference 
camera. ™s is called correspondence point problem in the stereoscopic view. 

At the time of retrieving (searching) correspondence point generally carr.ed 
out, imagepicl.-nppointn.exists, as is clear withreferencetoFIG.l,online where 

planedeterminedbylineofsightofthereferencecameraandangleofvisibilityofthe 

detectioncameraandimagepick-upplane(surface)ofthedetectioncameraintersect 



4 



(„oss) wUh each other. This line is called epipolar Una. Further, .hen positional 
relationshipbetweenthereferencecan.eraand.hedetectioneanteraandp— 

i„inrespec.vecar„eras(focaiaistanceorlength,etc.).e— ,itisposs.hle 

retrieval of correspondence points on this epipolar line. 

Anexamplewherecorrespondencepointofimagepick-uppointn,onp.cture 

i.agedhythereferenceca»eraisdetectedonpictureinragedhythedetectioncan,era 

.Ulnowbedescrihed. Atthistitne, as showninHG.2, with small arealOO around 

(correlative)valuesaredeterminedatseveralpointsontheepipolarlineofpict„reof 
the detection camera. In this case, resolution of the epipolar line, i.e., resolution of 

points n„ to n. correspond to distance numbers 1 to 6 corresponding to distances 
,ron. the reference camera. Further, these distance numbers 1 to 6 correspond to 
„on^elineofsightfromthereferencecamera.WhenI(x)isassumedtobe 

Wnance value ofpic^re imaged by reference camera and r(x-) is assumed to be 
luminance value of picture imaged by detection camera, correlation is calculated by 
the following formula (1). 

^\i{x^i)-rix'+i)\ -^^^ 
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1„ accordance with the formula (1 ), according as correlation becon.es larger 
(higher), degreeofcorrespondencebetweenpicturei^agedbythereference camera 

and picture imaged by the detection camera becomes higher, and accordtag as 
correlationbeco^ess^naller (low), thedegreeofcorrespondencebeconres lower. 

in addition, para„.eter corresponding to the degree of correspondence is assumed to 
— n value. The relationship between the evaluation value and respective 
l.age pick-up poh,tsn. to n^ontheepipolar line is showninFIG.3.Inthis case, 

according as correlation becomes smaller (lower), the evaluation value becomes 
larger, and according as correlation becomes larger (higher), the evaluation value 
becomes smaller. 

in accordance with this FIG. 3, image pick-up point n, correspondmg to the 
pointinwhichevaluationvaluebasedonthecorrelationcalculatedbytheformulad) 

is the lowest (mh,imum) is assumed to be correspondence point. In this case, the 
distance number is "3", Alternatively, while distance correspondh,g to the pomt 
where evaluation value is the minimum may be determined from respective :mage 

piclc-uppointsn,aspreviouslydescribed.therearealsoinstanceswhereinterpolatron 
betweensampleddataiscan-iedoutfromvaluesattheperipherywheretheevaluatron 
value is minimum to determine mh^hnum value. By determining distanceofpicture 

that the reference camera has imaged from the reference camera and the detection 
camerain this way.distancepictureis generated along withvariable-densitypicture 

having luminance information. 
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However, in the first technique for generating distance picture consisting of 
distance information and variable-density picture consisting of luminance 
information, etc. by cameraunit disposed at the above-described virtual position and 
in virtual direction, since such an approach is employed to develop (expand) object 
to be imaged into three-dimensional picture to generate distance pictures by using 
three-dimensional pictures obtained by carrying out developing (expansion) in a 
three-dimensional mamter. vast amount of calculations are required. Moreover, in 
this fnst technique, when three-dimensionalmeasurament is carriedout with respect 

toobject to be hnaged which has been imaged by plural camera units,it is necessary 
to image all coordinates which can be observed from the virtual camera unit, and 
three-dimensional measurements must be thus carried out from many view points. 
Further, in the first technique, it is also necessary to paste together pictures imaged 
by respective camera units when three-dimensional picture is generated. 

In addition, in the second technique which has been explained with reference 

to FIGS. 1 to 3, since such an approach is employed to shift picture imaged by the 
detection camera with picture imaged by the reference camera behig as template to 
retrievepicturetotherebygeneratedistancepicture,existenceofthereference camera 

is indispensable. 

Disclosure of the Invention 
This invention has been made in view of actual circumstances as described 
above, and its object is to provide a picture generating apparatus and a picture 
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generating method capable of generating, by only detection cameras, without using 
reference camera, distance picture indicating distance from the position where the 
reference camera is primarily disposed up to object to be imaged and variable-density 
(gradation) picture when object to be imaged is imaged from the reference camera. 

In order to attain the object as described above, a picture generating apparatus 
according to this invention comprises; two image pick-up means or more adapted 
for imaging (picking up image of) object to be imaged to generate picture data and 
respectively disposed at different positions; correlation detecting means for 
comparing, with each other, on epipolar line determined by connecting 
correspondence points of line of sight connecting virtual position and the object to be 
imaged and line of sight connecting positions of each of the imaging means and the 
object to be imaged, respectivepicturedatageneratedbytherespectiveimagepick-up 

means to detect correlation therebetween; and distance picture generating means for 
generating distance picture indicating distance between the virtual position and the 
object to be imaged on the basis of the correlation detected by the correlation 
detecting means. 

In accordance with such apicture generating apparatus, picture data generated 
bytwoimagepick-upmeansormore are used to compare, atthecorrelation detecting 

means, picture data generated by the respective image pick-up means to generate 

distance picture indicating distance between virtual position and object to be imaged 

at the distance picture generating means . 



A picture generating method according to this invention comprises: imaging 
(picking up image of) object to be imaged by two (solid-state) image piclc-np means 
(devices)ormore respectively disposed at differentpositionstogeneratepicture data; 

comparing, with each other, on epipolar line determined by connecting 
correspondence pomtsoflineofsightcomiectingvirtualposition and theobject to be 
imaged and lineof sight connectingpositionsofeachoftheimagepick-upmcans and 
the object to be imaged, respective picmre data generated by the respective image 
pick-up means to detect correlation therebetween; and generating distance picture 
indicating distance between virtual position and the object to be imaged on the basis 
of the detected correlation. 

In accordance with such picture generating method, such a procedure is taken 
tocomparepicturedatageneratedbytwo(solid-state)imagepick-up(imaging)means 

(devices) or more to detect correlation therebetween to generate distance picture 
indicating distance between virmal position and object to be imaged on the basis of 
the correlation. 

still ftirther objects of this invention and more practical merits obtainedby this 
invention will be more apparent from the embodiments which will be described 
below. 

Brief Description of the Drawings 
FIG. 1 is a view for explaining generation of distance information indicating 
distance betweenreference camera andobjectto be imaged byusingreference camera 
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and detection camera by the conventional technique. 

FIG, 2A is a view for explaining that small area around image pick-up point 
of reference camera is caused to undergo template and FIG. 2B is a view for 
explaining determination of correlation values by several points on epipolar line set 
on picture (on screen) of detection camera. 

FIG. 3 is a view showing the relationship between evaluation values and 
positions at several points on epipolar line. 

FIG. 4 is a block diagram showing an example of a picture generating 
apparatus according to this invention. 

FIG. 5 is a block diagram showing an example of calibration unit for carrying 
out calibration on the premise that distance picWre and variable-density picture are 

generated at virtual point A. 

FIG. 6 is a view for explaining an example for determining epipolar lines on 
respective detection pictures corresponding to respective coordinates of reference 
picture. 

FIG. 7 is a view showing observation points n„ and n^ observed at the picture 
surface of detection camera in the case where object to be imaged exists at distances 
Z] and Zj. 

FIG. 8 is a view for explaining an example of calibration carried out on the 
premise that distance picture and variable-density picture are generated at distance 
picture generating section in this invention. 
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FIG. 9 is a view for explaining an ex^nple of look-up Table generated by the 
calibration unit. 

FIG.lOisaflowchartshowinganexampleofproeessingwhendistancepicture 
and wiable-density picture are generated at distance picture generating section 
provided in the picture generating apparatus according to this invention. 

FIG. 11 is aviewfor explaining derivation ofpixels (n„ ton«) every distance 
number on epipolar lines of detection pictures corresponding to coordinate n, on 
picture that virtual camera should image. 

FIG. 12 is a view showing the relationship between evaluation value and 

distance number. 

FIG. 1 3 is a flowchart showing another example of processing when distance 
picture and variable-density picture are generated at distance picture generating 
section provided in thepicture generating apparatus according to this mvention. 

FIG. 14 is a flowchart showing an example of processing when evaluation 
value picture is generated at distance picture generating section provided in the 
picture generating apparatus according to this invention. 

Best Mode For Carrying Out the Invention 
Explanation will be given below in detail with reference to the attached 
drawings in comiection with an embodiment of this invention. 

This invention is applied to apicture generating apparatus (unit) 1 constituted 
as shown in FIG. 4, for example, and used when distance is measured to generate 
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distance picture. This picture generating apparatus 1 comprises plural detection 
cameras 3 (3a, 3b) for imaging (picking up image of) object 2 to be imaged, frame 
memories 4a, 4b into which picture data imaged by the plural detection cameras 3 are 
respectively stored, andadistancepicturegeneratingsectionSforgenerating distance 

picture and variable-density (gradation) picture by using picture data stored in the 
frame memories 4a, 4b. The picture generating unit (apparatus) 1 is adapted for 
imaging(pickingupimageof)theobject2tobeimagedbythedetectioncameras3a, 

3b to thereby generate distance picture and variable-density picture (hereinafter 
refeired to as "virtual picmre") to be imaged by a virtual camera 3c which is virtually 
disposed at virtual point A and does not exist. 

The detection cameras 3a, 3b comprise, e.g., CCD image sensor, etc. and serve 
to pick up image of object 2 to be imaged to thereby generate picture consisting of 
luminance information, etc. (hereinafter referred to as "detection picture") to store 
dataofthepicmre into the framememories 4a, 4b. Atthis time, the detection cameras 

3a, 3b respectively store the picture data thus generated into the frame memories 4a, 
4b every frame. 

Plural detection cameras 3a, 3b are disposed at positions different from the 
virtual point A in FIG. 4. In this example, detection cameras 3 are disposed at 
positions different from each other as the detection cameras 3a, 3b. It is to be noted 
that the positions where these detection cameras 3a, 3b are disposed and the number 
of cameras are not limited to the above, but they may be arbitrarily set. 
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The frame memories 4 (4a, 4b) respectively store thereinto pieture data from 
thedetectioncameras3a,3btooutputthemtothedistancepicturegeneratingsection 
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The distance picture gei,erating section 5 processes picture data from the frame 
memories 4a, 4b to thereby generate virtual picture generated at virtual camera 3c. 
Explanation will nowbe given with referencetothecalibrationunitlOofFIG. 

5 in ccmiection with calibration carried out on the premise that distance picture and 
variable-density picture at virmal point A are generated at the distance picture 
generating sectionSprovided in the above-describedpictoegeneratingunitlshown 

in FIG. 4. It should be noted that, in the explanation of the calibration unit 10. the 
same reference numerals are respectively attached to the portions similar to those of 
the above-described picture generating unit 1 and their detailed explanation will be 
omitted. 

The calibration unit (apparatus) 10 shown in FIG. 5 comprises a reference 
camera 1 1 disposed at the above-described virtual point A, and a frame memory 1 2 
into which picture data imaged by the reference camera 1 1 is stored, whereby picture 
imaged by reference camera 11 (hereinafter referred to as "reference picture") in 
which object2to be imaged is imaged byacalibrationsectionl3andpicture imaged 

by detection cameras 3a. 3b (hereinafter referred to as "detection picture") are used 

to carry out calibration. 

Thecalibrationsectionl3carriesoutprocessingtoallowthedetectioncameras 
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3a, 3b and the reference camera 1 1 to undergo positioning to determine epipolar line 
onthedetectionpicturetodetermineinadvancecorrespondencerelationshipbetween 
eoordinatesofreferencepictureanddetectionpictureto,e.g..storeitaslook-up table 

into the above-described distance picture generating section 5. 

This epipolar line is (straight) line where plane determined by optical centers 
(optical axes) of the reference camera 11 and the detection cameras 3a. 3b and 
observationpointn^ofthe reference cameralland(surfaceof)pictureofeachofthe 

detection cameras 3a. 3b intersect (cross) with each other. Now, explanation will be 
given in comrection with an example for determining epipolar line on each of 
detec,ionpicturescorrespondingtoeachofcoordinatesofreferencepicture.Itishere 

assumed that, for example, as shown in FIG. 6, point P on the plane placed within 
fl„ee-dimensional space is observed at point n, on reference picture of the reference 
cameralland is observed atpointn,on detection pictureofthe detection cameras. 

In the case where such a relationship holds, when projection (projective) transform 
matrix of 3x3 for carrying out transform processing from the point n, to the point n, 
is assumed to be H. the point „, of the detection picture is expressed as follows: 
nd = H*nb •••(2) 

Further, this distance picture generating section 5 implements a predetemtined 
projection transform processing to reference picture imaged bythe reference camera 

11 while changing parameters of this projection transform matrix H to determine 
parameters of the projection transform matrix H so that err or of luminance between 
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detectio„picturea„dreferencepicmrebe<x>mesmintaum.Bydeterminingprojecti^ 
transform matrix H, the calibration section 13 can retrieve, from on the detection 
picture, correspondence point corresponding to an arbitrary point of the reference 
picture. In .his example, as a method of detennining projection trartsform matrix H 
in which tmtsformed detection picture and reference picture are caused to be 
precisely in correspondence with each other (picture matching method), 
Levenberg-Marquardt minimum method (L-M method) is used. This calibration 
section 13 determines, every distance, projection transform matrix H by using such 
a technique to determine epipolar line by using the projection transfonrt matrix H. 

Namely, in this calibration section 1 3, as shown in FIG. 7, for example, when 
the above-describedL-Mmethod is used to determine projectiontransformmatrixH, 
with respect to distance Z,&omthe reference position, point n„ondetection picture 
corresponding to pointn, on the reference picture at distance Z,fi-om the reference 
position can be determined. In this case, since point n, on the reference picture is 
arbitrary, the calibration section 13 can calculate, as point n, on detection picture, 
correspondencepointscorrespondingtoallpointsn^onreferencepicturebyusingthe 

projectiontransformmatrixH, at distance Z, In a mamter similar to the above, the 
calibration section 1 3 determmes projection transform matrix H, in the case where 
plane to be observed is placed at distance Z, so as to become in parallel to plane 
placed at position of distance Zi- 

Further, the calibration section 1 3 calculates that the point n, on the reference 
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picture is projected onto respective points n„ and on the detection picture by 
projection transfonnmatrixH, and projectionfransfonntrtatrixH,correspondingto 

distaneeZ, anddistanceZ, inthecasewherethepointn^islocated at distance Z, and 
Z,. As stated above, the calibrationsectionl3inte.polatestwopointsofpointn„ and 
point n,, on the detectionpicture to calculate epipolar line to thereby have ability to 
determinetherelationshipofthe point n. on the referencepicture, the distanceZand 
the point n, on the detection pictare. By stormg such projection transform matrix H 
every distance Z. distance information of virtual picture to be generated by virtual 
camera 3c provided a, the picture generating unit 1 can be obtained by using the 
method which will be described later. It is to be noted that when interpolation is 
carried out in this way, it is necessary to set plane placed at distance positioned 
between the distances Z, and Z,.Itis to be forthernotedthatthe detail relatingto the 

example of calibration that the above-described calibration section 1 3 carries out is 

described in the Japanese Patent Application No. 20795 1/1997. 

Asamethod of determining distanceZfrom point n, on thereference picture 

and point n, on the detection picture, the following method may be employed. 
Namely, the calibration section 13 carries out calibration shown inHGS. 6 and 7 on 
calculated epipolar line to thereby determine the relationship between point n, on 
referencepicture and point n, on detection picture to store this relationship, e.g.,into 

look-up table. 

In this calibration, by using the fact that, e.g., as shown in FIG. 8, point at 
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which there intersect(cross)lineofsightconnecti„g point on the epipola^-lineof 
detection picture and optical center and hne of sight connecting point n, on the 
reference picture and optical center of the reference catrtera U corresponds to 
distance Z, from the reference camera 11 up to object to be imaged, distance Z, is 
caused to be distance number "1". This epipolar line is calculated by using the 
above-described projection transform matrix H. Moreover, point where there are 
intersect (cross) line of sight connecting point n. on the epipolar line of detectiort 
pictureandoptical center andlineofsightconnectingpointn^onthereferencepicture 

and optical center of reference camera 11 corresponds to distance Z, from the 
referencecamerallup to theobjecttobeimaged.Atthistime,distanceZ. is caused 

to be distance number "2". 

In this way, the calibration section 13 detects, as distance numbers 1 to 6, 
distances Z, to Z, corresponding to the points where lines of sight connecting points 
to n,, on epipolar lines of detection picture and line of sight connecting point n, 
on the reference picture and optical center of the reference camera 11 intersect or 
crosswitheachother successively with respecttopointsn,, to n,,ofdetectionpicture 

to thereby carry out calibration before distance picture generation carried out at the 
above-described picture generating apparatus 1. Further, the calibration section 13 
can obtain distance numbers 1 to 6 of pictures to be imaged at point n, on the 
referencepicture from the relationship between point n, on thereference picture and 

point n^ on the detection picture. 
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In addition, the calibration section 13 may set respective planes with respect 
to respective distances Z, to Z, to generate respective projection transform matrices 
indicating coirespondence relationship between poim n, on the reference picture and 
point on the detection picture, and, e.g., may set planes only with respect to three 
distances of distances Z„ Z,, Z, to determine projection transform matrk indicating 
correspondence relationship between pointn,onthedetectionpicfure and pointn, on 

the detection picture thereafter to determine, by interpolation, projection transform 
matrix with respect to distances Z„ Z, positioned between distances Z,. Z3, Z,. 

Namely, this calibration section 13 successively carries out, on the epipolar 
Ime, calibration of pomt on the detection picture corresponding to point n, on the 
reference picture from the optical center (optical axis) to thereby determme 
correspondence points n„ to n,, on the epipolar Une to thereby determine distances 
Z, to Ze to allow correspondence points n„ to n,, and distance numbers 1 to 6 to have 
correspondencerelationship therebetween to detennmetherelationship between point 

n, on the reference picture and point n, on the detection picture to prepare look-up 
table as shown m FIG. 9, for example. 

In this FIG. 9, there are stored, e.g., points (n„, n^, n,,, -) on detection picture 
that detection camera 3a located at distances (Z„ Z,, Z3, -) corresponding to a certain 
point n, existing on the reference picture has imaged and points (n^,, n,^, n„, -) on 
detection picture that detection camera 3b located thereat has imaged. Further, such 
look-up tables are generated, e.g.. every respective pixels on reference pictures, and 
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reference is made to pixels on the reference picture in the calibration section 13 as 
pixels on the viitual picture when distance picture is generated at the picture 
generating unit 1. 

Explanation will now be given with reference to the flowchart of FIG. 10 in 
connection with the example where, at the distance picture generating section 5 
provided in the picture generating unit (apparatus) 1, picture data generated at 
detection cameras 3a, 3b are used to generate distance picture and variable-density 
picture by making reference to look-up table generated in the above-described 
calibration unit 10. 

In accordance with the flowchart shown in FIG. 10, initially, at step STl, the 
distance picture generating section 5 initializes coordinate i of pixel of picture data 
of virtual picture of virtual camera 3c to thereby set i to zero (i=0). 

Subsequently, at step ST2, the distance picture generating section 5 initializes 
distance number j corresponding to resolution in the distance direction when 
calibration is carried out to set j to zero Q=0). Thus, the processing proceeds to step 
ST3 and step ST4. 

At the step ST3, as shown in FIG. 1 1, the distance picture generating section 
5 derives (generates) pixels n,, (i, j) (n,, to nj at distance number j on the epipolar 
line of detection picture corresponding to coordiante i (n^) on picture that the virtual 
camera 3c should image. Namely, when coordiante i on the virtual picture that the 
virtual camera 3c should image is determined at step STl, the distance picture 
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generating section 5 detects, from the look-up table, pixel n^^ (i, j) on the epipolar line 
of the detection camera 3a determined by the above-described processing. Thus, the 
processing proceeds to step ST5. It is to be noted that position of determined 
(detected) pixel n,, (i, j) of detection picture may be represented by sub-pixels in 
addition to integer value. 

At the step ST5, the distance picture generating section 5 extracts, as small area 
WA (i, j), the portion in the vicinity of pixel n^^ (i, j) at distance number j detennined 
at the above-described step ST3 with the pixel n^a (i, j) being as center. 

On the other hand, the distance picture generatmg section 5 carries out, at step 
ST4, also with respect to detection camera 3b, the processmg which has been 
explamed at the above-described step ST3. Namely, this distance picture generating 
section 5 detects, from the look-up table, pixel n.^ (i, j) at distance number j on 
epipolar line on detection picture that detection camera 3b has imaged corresponding 
to coordmate i on picture that the virtual camera 3c should image. Thus, the 
processing proceeds to step ST6. 

Also at step ST6, the distance picture generating section 5 carries out 
processing similar to the above-described step ST5 with respect to the detection 
camera 3b. Namely, the distance picture generatmg section 5 extracts, as small area 
WB (i. j), the portion in the vicinity of pixel n^^ (i, j) at distance number j determined 
at the above-described step ST4 with the pixel n^b (i, j) being as center. 

Then, at step ST7, the distance picture generating section 5 compares small 
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area WA (i,j) extracted at the above-described step ST5 and small area WB (ij) 
extracted at the step ST6 to thereby calculate correlation to obtain evaluation value 
sG). Here, the evaluation value sG) indicates correlation at the distance number j, and 
is calculated, e.g., by the following formula. 

Sl/U+0-i'U'+^-)l -(3) 

According as correlation becomes higher, it is indicated that (the degreee of) 
analogousness betv^een small area WA (i.j) and small area WB GJ) becomes higher. 
Moreover, as the result of the fact that the evaluation value sG) is calculated by the 
above-mentioned formula (2) in correspondence with correlation at the distance 
picture generating section 5, according as correlation becomes higher, the evaluation 
value has smaller value. Namely, this distance picture generating section 5 compares 
lummance pattern of small area WA GJ) of detection picture generated by the 
detection camera 3a and that of small area WB G,j) of detection picture generated by 
the detection camera 3b to thereby generate (the degree of) analogousness as 

evaluation value sG)- 

Then, at step ST8, the distance picture generating section 5 recognizes distance 
number j to thereby judge whether or not evaluation values sG) have been calculated 
with respect to all distance numbers. When it is judged that the section 5 has carried 
out processing with respect to all distance numbers j, the processing proceeds to step 
STIO. On the other hand, when it is judged that the section 5 has not yet carried out 
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processingwithrespectto all distancenumbersj,theprocessingpiWsto step ST9 

to increment distance number j. Thus, the processing proceeds to steps ST3 and ST4. 
Namely,thedistancepicturegeneratingsection5camesonttheprocessingexplained 

at the above-described steps ST3 to ST7 in comiection with all distance numbers j 
with respect to coordinates i of pixels of viitual picture of virtual camera 3c. 

Then, at step STIO, the distance picture generating section 5 selects evaluation 
value sG..) having minimum value of respective evaluation values s(i) in comiection 
with all distance numbers j with respect to coordinate i of virtual picmre. When 
evaluation values sG) corresponding to distance numbers 1 to 6 are obtamed at the 
distance picture generating section 5, as shown in FIG. 12, for example, evaluation 
value sGn. J having minimum value when distance number is "3" is obtained. At this 
time, the distance picture generating section 5 may interpolate belM^een respective 
evaluation values sG) obtained at the step ST7 to determine minimum evaluation 
value sG™J. The distance picture generating section 5 is operative so that when 
interpolation is made to obtam minimum evaluation value sG™„), the value in which 
the distance number n,„ is "3,3" is obtained as minimum evaluation value sQ^J. 
Accordingly, this distance picture generating section 5 obtains evaluation value sG„.i„) 
by interpolation to thereby permit accuracy of evaluation value to be high. 

Then, at step STl 1 , the distance picture generating section 5 stores, as distance 
information of coordinate i of distance picture imaged by the virtual camera 3c, 
distance Z corresponding to distance number j thatthe evaluation value sG™ J having 
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minimum value obtained at the above-described step STIO indicates. 

Then, at step STI2, the distance picture generating section 5 stores, as 
luminance infonnation at coordinate i of virtual picture, e.g., one value or both values 
of pixels of detection picture imaged by the detection camera 3a and detection picture 
imaged bythedetectioncameraSbcorrespondingto distance numberjthat evaluation 

value sO„J having minimum value obtained at the above-described step STIO 



indicates. 



Then, at step ST13, the distance picture generating section 5 carries out the 
above-described processing of steps ST3 to ST12 with respect to respective 
coordinates i on the virtual camera 3c to thereby judge whether or not processing for 
determming distance information and luminance information have been made with 
respect to all coordinates i of the virtual picture. Further, when the distance picture 
generating section 5 has judged that distance information and luminance information 
are obtained with respect to all coordinates i, it completes the processing. On the 
other hand, when the section 5 has judged that distance information and luminance 
information are not determmed with respect to all coordinates i, it mcrements value 
of coordinate i at step ST14. Thus, the processing returns to the step ST2 to carry out 
processing of steps ST2 to ST12 with respect to the incremented coordiante i+1 to 
repeat processing until distance information and luminance information are 
determined with respect to all coordinates i. 

The picture generating unit 1 provided with such distance picture generating 
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section 5 can generate distance picture indicating distance from viitual point A where 
camera is primai'ily disposed up to object 2 to be imaged and variable-density picture 
when the object 2 to be imaged is imaged from the virtual point A by only detection 
cameras 3a, 3b without disposing camera at virtual position with respect to virtual 
picture to be generated at the virtual camera 3 c. Accordingly, in accordance with this 
picture generating unit 1, it is possible to generate distance picture having distance 
informationwithrespecttorespectivepixelswhichcannotbe electronically generated 

as picture only by the virtual camera 3 c. 

Moreover, in accordance with the picture generating unit 1, when distance 
picture and variable-density picture from virtual camera disposed at virtual position 
are generated, camera is actually located (disposed) at the position from the virtual 
camerato carry outcalibration,therebymakingitpossibleto generate distancepicture 

and variable-density picture to be generated by the virtual camera without necessity 
to carry out complicated processing such that expansion is made into the coordinate 
system of the three-dimensional space of object to be imaged as in the prior art. 

Further, in accordance with this picture generating unit (apparatus) 1, in the 
case where, e.g., there exists screen for displaying picture at the front of user, even 
in the state where camera cannot be disposed at the front of user, plural detection 
cameras 3 are disposed at the periphery of user, whereby image pick-up operation is 
made from the front of user, thus making it possible to generate distance picture and 
variable-density picture to be generated. 
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Furthennore, in accordance with this picture generating unit (apparatus) 1, 
there may be disposed, at position of the virtual camera 3c, image pick-up camera of 
high fineness or camera of the type different from detection cameras 3a, 3b like 
camera using film, etc. Such picture generating unit 1 can pick up image of 
variable-density picture consisting of luminance information and color information 
independently of processing for measuring distance between virtual camera 3c 
disposed at virtual position and object 2 to be imaged, and can also measure distance 
of object 2 to be imaged which has been imaged by ordinary camera disposed at 
virtual position where virtual camera 3c is disposed. 

Explanation will now be given with reference to the flowchart shovm in FIG. 
1 3 in connection with another example where the distance picture generating section 
5 generates distance picture and variable-density picture by using picture data 
generated by detection cameras 3a, 3b. 

Initially, at step ST21, this distance picture generating section 5 initializes 
distance number j corresponding to resolution in distance direction when calibration 
is carried out to set j to zero G=0). 

Then, at step ST22, the distance picture generating section 5 initializes 
coordinates i of respective pixels on virtual picture of virtual camera 3c to set i to zero 
(i=0). Thus, the processing proceeds to step ST23 and step ST24. 

At the step ST23, the distance picture generatmg section 5 derives pixel n^X^, 
j) at distance number j on epipolar line on detection picture that detection camera 3a 
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has unaged corresponding to coordinate i on virtual picture that viitual camera 3c 
should image similarly to the above-described step ST3. Thus, the processing 

proceeds to step ST25. 

At the step ST25, the distance picture generatmg section 5 stores, as pixel Da(i) 
of parallax picture Da, luminance value of pixel n^^Ci, j) derived at the 
above-described step ST23. 

On the other hand, at the step ST24, the distance picture generating section 5 
derives, similarly to the above-described step ST4, pixel n,,(i, j) at distance number 
j on epipolar line on detection picture corresponding to coordinate i on virtual picture 
that virtual camera 3c should image. Thus, the processing proceeds to step ST26. 

Further, at the step ST26, the distance picture generating section 5 stores, as 
pixel D,(i) of parallax picture D„ luminance value of pixel n,,(i, j) derived at the 
above-described step ST24. 

As stated above, the distance picture generating section 5 carries out processing 
indicated by step S25 and step ST26 to thereby store pixels Da(i), Db(i) of parallax 
pictures Da, Db at distance number j of respective detection cameras 3a, 3b at 
coordiante i=0 of virtual picture. At this time, the distance picture generating section 
5 derives, by making reference to the look-up table, pixels Da(i), Db(i) corresponding 
to distance number j when areas on epipolar lines of respective detection pictures are 
calibrated by the calibration unit 10. 

Then, at step ST27, the distance picture generating section 5 judges whether 
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or not pixels Da(i), Db(i) of parallax pictures Da, Db are stored with respect to all 
coordinates i of viitual picture at distance number j. Further, when the distance 
picture generating section 5 has judged that pixels Da(i), Db(i) of parallax pictures 
Da, Db are stored with respect to all coordmates i, the processing proceeds to step 
ST29. On the other hand, when the section 5 has judged that such pixels are not 
stored with respect to all coordiantes i, the coordiante i is incremented at step ST28 
so that coordinate i+1 result. Thus, the processing returns to the step ST23 and the 
step ST24 for a second time. Namely, the distance picture generating section 5 
repeats processing of step ST23 to step ST28 until pixels Da(i), Db(i) of parallax 
pictures Da, Db are stored with respect to all coordinates i at distance number j. 

At step ST29, the distance picture generating section 5 compares, by using 
above-mentioned formula (1 ), pixel Da(i, j) included in parallax picture Da and pixel 
Db(i, j) included in parallax picture Db which are obtained by carrying out the 
above-described processing of steps ST23 to ST28 to thereby determine correlation 
between detection picture imaged by the detection camera 3 a and detection picture 
imaged by the detection camera 3b to determine evaluation value s(i, j) corresponding 
to correlation with respect to each pixel. Further, the distance picture generating 
section 5 generates evaluation value picture S(i, j) consistmg of evaluation value s(i, 
j) of coordinate i at distance j. It is to be noted that the detail with respect to 
processing for generating this evaluation value picture S(i, j) will be described later. 
Then, at step ST30, the distance picture generating section 5 recognizes 
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distance number j to thereby judge whether or not evaluation value pictures S(i, j) 
have been generated with respect to all distance numbers j. When processing have 
been carried out with respect to all distance numbers j, the processing proceeds to step 
ST32. On the other hand, when processing have not been carried out with respect to 
all distance numbers j, the processing proceeds to step ST31 to increment distance 
number j to allow the distance number to be j+1 to repeat processing of steps ST23 
to STB 1 for a second time. 

At the step ST32, coordinates i of respective pixels on virtual picture of the 
virtual camera 3c are initialized so that coordinate i=0 results. 

Then, at step ST33, the distance picture generating section 5 retrieves value 
where evaluation value s(j) which is value of evaluation value picture S(i, j) every 
distance number j at coordiante i obtained by the above-described processing is 
minimum to derive (generate) distance number j^j„ corresponding to pixel s(i, j^i„) 
having minimum evaluation value. In addition, this distance picture generating 
section 5 may interpolate respective evaluation values sQ) every respective distance 
numbers j to determine minimum evaluation value s(in,i„)- As stated above, the 
distance picture generating section 5 determines, by interpolation, minimum 
evaluation value sQ^J, thereby permitting accuracy of the evaluation value sG) to be 
higher. 

Then, at step ST34, the distance picture generating section 5 stores, as distance 
information of coordinate i of distance picture imaged by the virtual camera 3C, 
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distance Z corresponding to distance number that the minimum evaluation value 
s(i, jmi„) obtained at the above-described step ST33 indicates. 

Then, at step S35, the distance picture generating section 5 stores, as luminance 
information at position i of virtual picture, e.g., one value or average (mean) value of 
both values of luminance information of pixels of detection picture imaged by 
detection camera 3a and detection picture imaged by detection camera 3b 
corresponding to distance number that the minunum evaluation value s(i, u J 
obtained at the above-described step ST33 indicates. 

Then, at step ST36, the distance picture generating section 5 cames out, with 
respect to respective positions i of pixels on the virtual camera 3c, the 
above-described processing of step ST33 to step ST35 to thereby judge whether or 
not processing for determining distance information and luminance information are 
carried out with respect to all coordinates i of virtual picture. Further, when it is 
judged that distance information and luminance information are determined with 
respect to all coordinates i, the distance picture generating section 5 completes the 
processing. On the other hand, when it is judged that distance information and 
luminance information are not determined with respect to all coordinates i, the 
coordinate i is incremented at step ST37. Thereafter, the processing returns to the 
step ST 33. Thus, the distance picture generating section 5 carries out processing of 
step ST33 to step ST36 with respect to the incremented coordmate i+1 to repeat 
processing of steps ST33 to step ST37 until distance information and luminance 
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information have been determined with respect to all coordinates i. 

AnexaiTipleforgeneratingevaliiationvaluepictureS(i,j)fromparallaxpicture 

Da and parallax picture Db at the above-described step ST29 will now be described 
with reference to FIG. 14. 

At the time of generating evaluation value picture S(i, j), the distance picture 
generating section 5 initializes, initially, at step ST41, coordinate i of picture of 
evaluation value picture S(i, j) into zero (0). 

Then, at step ST42, the distance picture generating section 5 compares 
luminance patterns of small area in the vicinity of parallax picture Da(i) and small 
area in the vicinity of parallax picture Db(i) to thereby calculate evaluation value sQ) 
every small area. As a result, there exists evaluation value sQ) corresponding to 
distance number j as information of each pixel. At this time, the distance picture 
generating section 5 may calculate evaluation value s(i) by using the 
above-mentioned formula (1). 

Then, at step ST43, the distance picture generating section 5 stores, as pixels 
of evaluation value picture S(i, j), evaluation values obtained by calculation at the 
above-described step ST42. 

Then, at step ST44, the distance picture generating section 5 judges whether 
or not pixels of evaluation value picture S(i, j) are generated with respect to the 
entirety of picture on screen. When it is judged that storage has been carried out as 
pixels of evaluation value picture S(i, j) with respect to all pixels from coordinate i 
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of viitual picture, the processing is completed. When it is judged that storage has not 
yet been carried out as pixels of evaluation value picture S(i, j) with respect to all 
pixels from coordinate i, the coordinate i is incremented into coordinate i+1. The 
section 5 carries out processing of step ST42 and step ST43 for a second time to 
repeat processing until pixels of evaluation value picture S(i, j) are stored resultantly 
with respect to all coordinates i. 

As stated above, the picture generating unit 1 can generate distance picture 
indicating distance from the position where camera is primarily disposed until object 
2 to be hnaged and variable-density picture when the obj ect 2 to be imaged is imaged 
from virtual camera 3c by only detection cameras 3a, 3b without disposing camera 
at virtual position with respect to virtual picture to be generated at virtual camera 3c. 

Further, thispicturegeneratingunit(apparatus) 1 compares luminance patterns 
of small area in the vicmity of parallax picture Da(i) and small area in the vicinity of 
parallax picture Db(i) to thereby calculate evaluation values sQ) every small areas to 
calculate coordinates of virtual picture while successively incrementing such 
coordinate to thereby generate evaluation value picture S(i, j) every respective 
distance numbers to select minimum evaluation value sG.J to generate distance 
picture consisting of distance mformation and variable-density picture consisting of 
luminance information. Thus, quantity of calculations can be reduced as compared 
to the above-described processing shown in FIG. 10 or FIG. 12. 

It is to be noted that while, in the processing of the distance picture generating 
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section 5 wliich has been explained with reference to the above-described flowcharts, 
the example for determining distance information and luminance information with 
respect to coordiantes i of the entirety of viitual picture has been described, distance 
information and luminance information may be determined with respect to a portion 
of coordinates i of virtual picture. 

Moreover, at the time of calibration carried out on the premise of the 
processingofthe distancepicturegenerating sections whichhas been explained with 

reference to the above-described flowcharts, such an approach may be employed to, 
e.g., project pattern light onto the plane to be imaged to thereby paste texture onto the 
objectto be imaged. As stated above, the picture generating unit 1 pastes texture onto 
object to be imaged at the time of calibration, thereby making it possible to improve 
accuracy of calibration to improve accuracy of distance measurement. 

In addition, in order to improve accuracy of distance measurement, this picture 
generating unit (apparatus) may comprise: camera for projecting, e.g., pattern light 
of infrared area onto object to be imaged, disposed at position where virtual camera 
3c is disposed and including infrared ray shielding filter; and detection cameras 3 
positioned therearound. At this time, camera in which calibration has been carried 
out is used as it is as virtual camera. Thus, this picture generating unit (apparatus) 1 
receives pattern light of infrared area by the detection cameras disposed therearound 
to generate distance picture, and to generate variable-density picture by camera 
disposed at the position where virtual camera 3c is disposed. At this time, picture 
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generated by camera disposed at the position where virtual camera 3c is disposed is 
utilized as variable-density picture as it is. 

Industrial Applicability 
As explained above in detail, the picture generating apparatus accordingto this 
invention comprises: correlation detecting means for comparing, with each other, on 
epipolar line determined by connecting coiTCspondence points of Ime of sight 
comiecting virtual position and object to be imaged and line of sight connecting 
position of each image pick-up means and the object to be imaged, respective picture 
data generated by the respective image pick-up means to detect correlation 
therebetween; and distance picture generating means for generating distance picture 
indicatmg distance between virtual position and object to be imaged on the basis of 
correlation detected by the correlation detecting means, whereby picture data 
generated by two image pick-up means or more are used to compare, at the 
correlation detecting means, picture data generated by the respective image pick-up 
means to detect correlation therebetween. Accordingly, it is possible to generate, 
without disposing camera at the virtual position, distance picture indicating distance 
from the virtual position to the object to be imaged and variable-density picture when 
the object to be imaged is imaged from the virtual position. 

In addition, the picture generating method according to this invention 
comprises: comparing, with each other, on epipolar line determined by connecting 
correspondence points of line of sight connecting virtual position and object to be 
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imaged and line of sight comiecting position of each image pick-up means and the 
objecttobeimaged,respectivepicturedatageneratedbytherespectiveimagepick-up 

means to detect correlation therebetween. Accordingly, it is possible to generate, on 
the basis of the detected correlation, distance picture indicating distance from virtual 
position to object to be unaged and variable-density picture when the object to be 
imaged is imaged from the virtual position. 
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CLAIMS 

1 . A picture generating apparatus comprising : 

two image pick-up means or more for imaging (picking up image of) an object 
to be imaged and respectively disposed at different positions; 

correlation detecting means for comparing, with each other, on epipolar line 
determined by comiecting correspondence points of line of sight connecting virtual 
position and the object to be imaged and line of sight connecting position of each of 
the image pick-up means and the object to be imaged, respective picture data 
generated by the respective image pick-up means to detect correlation therebetween; 
and 

distance picture generating means for generating distance picture indicating 
distance between the virtual position and the object to be imaged on the basis of the 
correlation detected by the correlation detecting means. 

2. A picture generating apparatus as set forth in claim 1 , 

wherein the correlation detectmg means compares, with each other, picture data 
of small areas consisting of plural pixel data positioned on the epipolar line to detect 
correlation therebetween. 

3 . A picture generating apparatus as set forth in claim 1 , 

which comprises variable-density (gradation) picture generating means for 
generating variable-density picture when the object to be imaged is imaged from the 
virtual position on the basis of respective picture data generated by the respective 
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image pick-up means, 

wherein the variable-density picture generating means generates 
variable-density picture by using luminance information of picture data generated by 
the respective image pick-up means. 

4. A picture generatmg apparatus as set forth in claim 1 , 

which comprises a reference camera disposed at the viitual position, 
wherein the reference camera generates variable-density picture of the object 
to be unaged, and 

wherein the distance picture generating means generates distance picture 
indicating distance between the reference camera and the object to be imaged on the 
basis of picture data generated by the two image pick-up means or more. 

5. A picture generating apparatus as set forth in claim 4, 
which comprises: 

light emitting means for irradiating pattern light of a predetermined area onto 
the object to be imaged; and 

filter means for shielding pattern light of the predetermined area incident to the 
reference camera, 

wherein the reference camera generates variable-density picture of the object 
to be imaged, and 

wherein the distance picture generating means generates distance picture 
indicating distance between the reference camera and the object to be imaged on the 
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basis of picture data that the two image pick-up means or more generate by using Hght 
reflected from the object to be imaged onto which the pattern light is irradiated. 

6. A picture generating apparatus as set forth in claim 4, 

wherein the reference camera is used for generating distance data indicating the 
relationship between pictures imaged by the respective image pick-up means and 
distance between the virtual position and the obj ect to be imaged. 

7. A picture generating apparatus as set forth in claim 1 , 

wherein the correlation detecting means compares, with each other, picture data 
on each epipolar line corresponding to distance between the viitual position and the 
object to be imaged to detect correlations every respective distances with respect to 
respective pixel blocks consisting of at least one pixel constituting distance picture, 
and 

wherein the distance picture generating means allows distance in picture data 
having highest correlation of correlations every respective distances detected by the 
correlation detecting means to be distance with respect to the object to be imaged of 
pixel block of the distance picture. 

8 . A picture generating apparatus as set forth in claim 7, 

which comprises variable-density picture generating means for generating 
picture data when the obj ect to be imaged is imaged from the virtual position by using 
plural picture data imaged by the respective image pick-up means on epipolar line 
corresponding to distance having highest correlation. 
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9. A picture generating apparatus as set forth in claim 1 , 

wherein the correlation detecting means carries out, with respect to all 
distances, processing to compare, with each other, picture data indicating a 
predetermined distance of plural picture data on each of the epipolar lines 
corresponding to distance between the virtual position and the object to be imaged to 
detect correlation in regard to the predetermined distance with respect to the entirety 
of the distance picture, and 

wherein the distance picture generating means allows distance in each of 
picture data having highest correlation every respective distances every respective 
pixel blocks consisting of at least one pixel constituting respective distance pictures 
to be distance with respect to the object to be imaged in regard to each of pixel 
blocks. 

10. A picture generating apparatus as set forth in claim 9, 

which comprises variable-density picture generating means for generating 
picture data whentheobjectto be imaged is imagedfromthevirtualpositionby using 

plural picture data imaged by the respective image pick-up means on the epipolar line 
corresponding to distance having highest correlation. 

11. A picture generating method comprising: 

imaging (picking up image of) an object to be imaged by two (soHd-state) 
image pick-up means (devices) or more respectively disposed at different positions 
to generate picture data; 



38 

comparing, with each other, on epipolar Hne determined by connecting 

correspondence points of line of sight comiecting viitual position and the object to be 

imaged and Ime of sight comiecting position of each of the image pick-up means and 

the object to be imaged, respective picture data generated by the respective image 

pick-up means to detect correlation therebetween; and 

generating distancepictureindicatingdistance between virtual positionandthe 

object to be imaged on the basis of the detected correlation therebetween. 

12. A picture generating method as set forth in claim 1 1 , 

wherein picture data of small areas consisting of plural pixel data located on 
the epipolar line are compared with each other to detect correlation therebetween. 

13. A picture generating method as set forth in claim 1 1 , 

wherein luminance patterns of respective picture data generated by the 
respective (solid-state) image pick-up means (devices) are used to generate 
variable-density (gradation) picture. 

14. A picture generating method as set forth in claun 1 1 , 
which comprises: 

comparing, with each other, respective picture data imaged by the (solid-state) 
two image pick-up means (devices) or more to detect correlation of the entirety of 
picture; and 

generating distancepictureindicatingdistance between thevirtualpositionand 
the object to be imaged on the basis of the correlation of the entirety of the picture. 



39 

15. A picture generating method as set forth in claim 1 1 , 
which comprises: 

comparing, with each other, picture data on each epipolar line corresponding 
to distance between the virtual position and the object to be imaged; 

detecting correlations every respective distances with respect to respective 
pixel blocks consisting of at least one pixel constituting the distance picture; and 

allowing distance m each picture data having highest correlation of correlations 
every respective distances to be distance between the virtual position and the object 
to be unaged. 

16. A picture generating method as set forth in claim 1 5, 

wherein plural picture data imaged by the respective (solid-state) image 
pick-up means (devices) on epipolar line corresponding to distance in each of the 
picture data having highest correction are used to generate picture data when the 
object to be imaged is imaged from the virtual position. 

17. A picture generating method as set forth in claim 1 1 , 

wherein a procedure is taken to carry out processing with respect to all 
distances to compare, with each other, picture data indicating a predetermined 
distance of plural picture data on each epipolar line corresponding to distance 
between the virtual position and the object to be imaged to detect, in connection with 
the entirety of the distance picture, correlations with respect to the predetermmed 
distance, and 
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wherein a further procedure is taken to allow distance in each of the picture 
data having highest correlation of correlations eveiy respective distances, every 
respectivepixelblocksconsistingofatleastonepixelconstitutingrespectivedistance 

pictures, to be distance with respect to the object to be imaged in regard to each of 
pixel blocks. 

18. A picture generating method as set forth in claim 1 7, 

wherein plural picture data imaged by (soUd-state) image pick-up means 
(devices) on epipolar line corresponding to distance in each of the picture data having 
highest correlation are used to generate picture data when the object to be imaged is 
imaged from the virtual position. 



41 

ABSTRACT 

A picture generating apparatus according to this invention comprises: two 
image pick-up devices 3a, 3b or more adapted for picking up image of an object 2 to 
be imaged to generate picture data and respectively disposed at different positions; 
a correlation detecting section 5 for comparing, with each other, respective picture 
data generated by the respective image pick-up devices 3 a, 3b on epipolar line 
determined by connecting correspondence points of line of sight connecting virtual 
position A and the object 2 to be imaged and line of sight connecting position of each 
of the image pick-up devices 3a, 3b and the object 2 to be imaged to detect correlation 
therebetween; and a distance picture generating section 5 for generating distance 
picture indicating distance between virtual position and the obj ect to be imaged on the 
basis of correlation detected by the correlation detecting section 5. 
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